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Recent educational policy changes at the federal, 
state, and local levels, coupled with the current economic crisis, 
will have a significant impact on the nature of program evaluation 
and on the emphasis placed upon it, according ^ the authors of this 
report. The document describes the evaluation requirements contained 
in the three chapters of the fed^r^l Education Consolidation and 
Improvement Act and asserts upon analysis of the law that needs 
assessemnt, program implementation evaluation, descriptive 
evaluation, management evaluation, cost effectiveness, outcome 
evaluation and sustained gain evaluation components should be 
incorporated into state and local education agencies' programs. 
Asstuning that the costs and . information yields of these processes 
will determine their value, the authors assess th«^ likelihood of 
their adoption under various conditions. After noting some inajor ^ 
trends in evaluation, including increased utilisation of eva*Luation^ 
and qualitative methodology, and a tendency to adapt methods from 
such disciplines as journalism, .philosophy, and business, the 
doctunent concludes with predictions regarding evaluation policy and 
practice during the 1980 *s. These predictions include reduced 
-emphasis on large-scale sunimative evaluations and program 
evaluations, tailure of the trend toward qualitative evaluation, 
increased use of aucliting methods, and a growing emphasis on ne.w 
electronic technology. (Author/PGD) 
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INTRODUCTION ^ ' 

That many changes are occurring in the field of education, is self 

evident. In the recent past the entire educational establishinent/has 

been subjected to strong challenges To its i^ilosophy, economics, policy 

and organizational structure. These challenges havfe resulted in pfofound 

changesr from the highest levels of organization down to the classroom. 

At the federal level we have seen the creation of a cabinet level 

Department of Education and its proposed abolishment. We have widiessed 

massive cuts in the allocation^ for federal programs designed to serve 

the educational needs of diverse special populations, with promises af 

even further|cuts in the future. We have experienced sweeping policy 

reform as the "new federalism" has been installed, manifesting itseff 

through* block grant legislation throughout the human^services and • 

educational fields. To compound matters, all of these changes are 

occurring during an era widely acknowledged as the worst economic crisis 

period since the great depression of the 1930s. Rampant inflation 

4 

coupled with a deepening recession has created severe financial problems 
for school districts and state education departments who must depend 
primarily on local taxation for the resources to operate their -systems. 
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Woven into the fabric of the new political philosophy is a strong 

thread of fiscal ccyiser vatism that demands increased fiscal ♦ 

^^^^^^accourttability . >aced with the dual threat of inflation and. reduced 

l^¥e*g^ of ^trfi^ing, legislators and policy makers at local, -state and 

federal levels have an increased 'need for accuraj^ and timely information 

about the'- costs and effectiveness of the educatio^l^programs for which 

they are providing support. ' , ' * 

Since the advent of large scale feder^ assistance pragrams in ^ 

* 

J ediication in the mid-sixties, evaluation has played a. major role irj/ 

providing information to policy makers regarding program effectiveness. 

The history of educational program evaluation however, has been 

characterized by f?iilure to live up ,to the* expectations many held for 

it. For a variety of reasons, too numerous and complex to discuss here, 

evaluation has not proyen effective as a major tool, for the improvement 

* • 

of education. One of the commonly cited problem 'areas is lack of 

• ^ ** 

utilipZation of the evaluation studies that are conducted. We will - 

examine this issue more closely in a s\ibsequent section of this paper. 

\ ■ 

EFFECTS OF FEDERAL Li:GISLATION ON * 
EDUCATIONAL Ev'aLUATION POLICY AND PRACTICE ^ 

Changes in educational legislation are ha'ving a profound effect on 

educational policy at state and local levels. An examination of the 

effects of the Education Consolidation and Improvement Act of 1981 (JSCIA 

or Public Law 97-35) can provide us with some basils ,for predicting the 

types of evaluation activities we might expect to see in the near 

y 

future. This law, consisting of three chapters, contains new features 
regarding evaluation and reporting requirements for LEAs andu,SEAs. 
Chapter I is a somewhat simplified version of the old Title I retaining 

2 



tli^ basic intent and program stru):ture of its pre<jecessor . Chapter II 
consolidates 29 other categarical aid programs into one Block Grant. 
Chapter III contains "general provisions" pertaining to Chafrters I and 
II. One stated intent of the new law is to "reduce the, administrative 
and paperwork burden" therefore the requirements for maintenance of 
effort, supplement — not supplant, and comparability of services" of the 

Jold ^tle I are simplified for Chapter I.* Before p^sage of P.L. 97-35, 
siks were required to follow federal formulas and regulations' regarding 
allocation of funds, student selection, reporting requirements, and in 
the case of Ti^le I, evaluation methodolbgy. Under the new law, 

^ responsibility for determining educational needs and setting priorities 
for allocations, arfd for defining, imj^ementing and evaluating their 
programs wiThin^^^e block grant, will be Shifted to, the LEA level. In 

. addition, LEAs will exercise control ovej^ at l^ast' 80 percent of the 
funds all^ated under Chapter II. 

\ 

The law contains specific evaluatipn and reporting requirements for 
^LEAs and sLas.- In addition to the^e explicit needs, there are other 
evaluative, record keeping and reporting needs implied ab the local and 
state- levels . * . ' 

Information and Reporting, Needs Under P.L. 97-35 

^ Local Needs * ' 

Each LEA is required to file an application with the SEA for a*period 

not to exceec^ three years, but subject to annual updating. The LEA 

application has four basic components, (1) allocation plan (t^ include 
* # 

private schools), (2) compliance assurances (pLncluding e<iuitable r 
participation of private; schools), (3) agreement to keep r^ords for 
fiscal audit and program evaluation as required by state, (4) provision 



for systematic consultation with parent^^ administrative personnel and . 
Other appropriate gfoups. Sub-chapter A of Chapter II provides for the 
improvement of basic skills- instruction. This sub-chapter contains 
specific r^equirements for diagnostic assessment, testing and sc^yxoi level 
evaluation. It also authorizes activities aimed at improving the 
planning, management and* implementation o^ educational programs. 

State Needs ' 

Each state education agency is also required to submit an application 
at least trienniajly, 4which may be amended annually), containing Severn 
components: (1) designation of the S|)A as the agency responsible for 
administration and supervision of the consolidated programs; 

(2) establishment of an advisory council, appointed by the 'Governor to 

advise the SEA on allocation of funds for state functions (20% maximum), 

formula for allocation of funds to LEAs (80% minimum), planning, 
> 

development, sup^^^rt, implementation and evaluation of state programs? 

(3) detailed allocation plan for the 20 percent state share including 

private schools and administration; (4) prov^ision for public notice and 

f 

dissemination of all information regarding points included in (2) and (3) 
'above; (5) provision for ^miual evaluation and public reporting of the 
^ef f ectiveness of consolidated programs beginning in 1984; ^6) provision 
%oi keepiJig records and infor-»§tion "required by the Secret'alry" for 
fiscal audit and program evaluation; (7) assurances of compliance with 
consolidation requirements. ' 

. ' - " * 

Implicit, Requirements and Needs 

While the intent of P.L. 97-35 is to reduce the administrative and 
paperwork burden on the LEAs and SEAs, * there is still a substantial 
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" requirement for collepting, organizing, interpreting and reporting 

t * , 1 ' t 

information. Whereas'in the past federal regulations have stipulated 

more specif icail^, the type' of information required, the evaluation 

specifications of the >new law will not be enforced by regulation. 

However, draft regulations of a fairly general type have been published. 

The evaluation r^quirem^t for Chapter I is greatly reduced in the new 

regulations. It is that 

. , "An LEA ... shall, at least once every three 

years, conduct an evaluation of its Chapter I project 
that Includes 

(a) Objective Measurements of educational 
achievement in basic skills, and , 

(b) A determin-ation of whether improved 

performance is sustained over a period more ^ 
than one year. (Section 200.54) 

A 

* There are no specific evaluation or reporting re<juirements stated f.or the 
SEA. A handbook is being prepared for SEAs and LEAs containing 
nonbinding guidelines for areas not specifically covered by the -r^ 
^regulations. 

The lack of specific binding evaluation regulations and the need for 
\systematic data collection andjj^reporting can be viewed as either 
advantageous or disadvantageous as a matter of perspective. One 

\ - . \ * 

disadvantage is that SEAs m^y have to define their own roles withou^ the 
federal guidance they hqive come to depend on'. On the advantageous>,side,. 
th^e* features have two^major benefits: (1) they accomplish the desired 
result of reducing the federally required paperwork burden and (2) they 
•can provide SEAs and LEAs with a new flexibility and create an* 
opportunity 'structure for the application of newly emerging methodologies 
in evaluation and reporting. At the local level this flexibility is 
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accompanied by new responsibilities for determining educational needs and 
establishing program prior ities *based^ on those needs. 



The Council of Chief State School Of ficersj Response 

A special task force was formed by the Council of Chief Stafe School 
Officers (CCSSO) Committee on Evaluation and information Systems (CEIS)," 
to review the evaluation requirements of ^the new law and make^ 
recommendations to the CCSSO regarding the types of evaluation most 
appropriate for Chapters I and II. The task force met on December 18, 
198| and issued an, interim report in which they recorded their beliefs' 
regarding ^valuation, ^he CEIS task force formulated five "beliefs" 
regarding ECIA-81, which in their vifew should undergird any evaluation 
efforts. , ^ * . 

1. The evaluation efforts should be appropriate to the scope and 
nature of the activity being 'evaluated.^ Chapter I being larger 
and more focused lends itself well to a summative, aggregative 
approach. Chapter II. on the oth^er hand, is much smaller and 
more diverse, requiring, therefore, ^ variety of approaches. 

2. The evaluation appro^che^ taken should be suggestive and not 
required for the states. Every effort should be made to 
encourage LEAs to" collect data commensurate with the needs of 
the SEA. 

* 

3. The SEAs and. LEAs should make every effort to collect those data 
for programs supported by both chapters which will insure that 
thk federal funds have been well spent and that the states have 

^ been accountable.* , 

4. The overall methodology of the Title I evaluation and reporting 
system ("^TIERS) is adequate and should be maintained for 
Chapter I with some modifications. 

\ 

5. The evaluation of Chapter II should addre^^s two questions 
immediately: 

1. How effectively has the spirit of the block granting been 
implemented at state and local levels, and (2) to what 
extent does the block grant^ funding meet the criterial 
needs approj^riate to Chapter II which have been identified 
by the states and local sch^aol districts? ... A subsequent 
evaluation question would deal with the educational 
impact ... 

\ 



« 



The CEIS task forde produced three, separate sets of suggestions for 
the evaluation of Chapter II, one each for impact evaluation , evaluation 
of congruence between Chapter II programs and SEA/I^EA neecjs and 
evaluation of Chapter II implementation , Thpse three areas would appear 
to foreshadow the types of evaluation activity deemed most appropriate by 
this important group of state level eval^ators. 



Emerging Trends in Evaluation 

Our analysis of the ,new law combined with the CEIS task force views, 

reveal "to us a set of emerging needs in the^area of evaluation. 

o LEAs will need to identify their own local educational needs and 
establish priorities for funding based on those needs. ( Needs 
assessment) 

o LEAs will need tx) learn how to conduct evaluation studies that -i^ 
address questions regarding the operation of pro*grams for 
purposes of making in-course corrections. ( Implementation 
evaluation ) 

o LEAfi and SEAs will need to develop systems for c^ollecting, 

maintaining , and retrieving data on p^rogram costs, ' participation 

levels and extent. ( Descri ptive evaluation) 

^^v 

o LEA and SEA evaluators will need to learn to work more directly 
and effectively with project managers and decision makers in a 
helping-consulting role to assure that programs proceed in a way 
consistent with their objectives. ( Management evaluation ) 



o LEAs and SEAs^ will need to more effectively delineate the 
relationship of program costs to 'program outcomes. ( Cost 
ef f ectivness ) 

o LEAs and SEAs will continue to need to evaluate program 
outcomes . ( Impact or outcome evaluation ) 

o LEAs will need to report on their success in maintaining 
improved performance. ( Sustained gains ) 

The extent to which any particular evalua^tive data need or 

combil^iation of needs presents itself, will be in part a function of SEA 

policy malcing with respect to the new law and its interpretation. Our 

experience in providing technical assistance in evaluation has brought us' 
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in contact with each of these needs and has led to the development of 

methods and materials for the provision of support to SEAs and LEAs in 

» 

these need areas. 

Recently, the TACs have begun to" combine techniques from other 
disciplines ^such as policy analysis, management consulting and cost 
analysis with traditional and nontraditional evaluation techniques to 
expand the scope and increase the effectiveness of technical assistance 
efforts ia evaluation and reporting. This trend has begun to emerge in 
direct response tC^A and LEA concerns over the lack of relevancy and 
utility of some of the more traditional approaches. 

4 

Types of Evaluation Most Applicable to SEA and LEA Needs 

Our experience in providing technical assistance in support of TIERS 
has taught us that the traditional concept of impact evaluation while 
valuable for determining project effectiveness in a global sense, is 
severely limited in terms of serving the continuous, decision making 
needs of educators. Program evaluation based on outcome measures such as 
achievement tests, can* provide gross level information about the overall 
effectiveness of a program, but is of limited value in terms of making 
program improvements. For this reason, we haVe focused our attention 
more recently on the concept of evaluating program implementation. In 
this approach classroom teachers, project leaders and school building 
principals are being trained in methods of documentation, data 
^collection, interpretation and application of results to improve the 
education process as it is taking place. 

We are also now bringing evaluation directly into the classroom by 
training teachers 'in techniques of pupil assessment and classroom 
observation for time-oh-task monitoring. These techniques are expected 



8 



ERIC 



to have direct benefits to the teachers^^ijy^ classroom 

management • By providing teachers with evaluative tools over which they 

have direct control, we hope to create a more direct link between 

evaluation and instruction, thus bringing evaluative tec^hry.ques to bear 

on the instructional process at the most effective point in the process. 

Cost analysis will likely become a critical feature of evaluation 

^under the new legislation. This is obviously so, because of the 

increased emphasis on fiscal accounting as mentioned earlier. We have 

begun developi-ng cost analysis techniques and working with ^EAs and LEAs 

in their application. In this effort we are drawin^g on the works bf 

Henry Levin at Stanford, as wpll as others. Much remains to be done in 

I 

this area but the interest level ampng SEAs is high and there is some 
promise of future pxayoff for current efforts. • 

Advanced technology, and in particular the Microcomputer, affords 
education a powerful tool for the collectio^n^ storage, manipulation and 
retrieval of data for more effective decision making. In the Technical 

Assistance Centers for Title I Evaluation (TAC) we have gained valuable 

\ 

experience in developing data basQS and jEijipil^J^ormation systems using 
microcomputers, for LEA clientele. We are also working on using 
microcomputers with optical scanning equipment to process survey data for 
needs assessments and similar evaluative survey work. 

Matching audience information needs and technical competencies with 
appropriate reporting^ and data display techniques ' is a constant challenge 
to evaluators. The requirement for public reporting of evaluation 
results by the LEA and sharing of information with the Governor's 
advisory council by the SEA will tend to emphasize the need for adaptive 
reporting techniques matching the audience information needs with ^ 

9 
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appropriate data prtsentation methods. The TACs have developed Workshops 
and instructional mi^^^ls on how to identify the most appropriate 

reporting method and how to,preent the necessary information to each 

i 

audience most effeptively. This experience can provide the basis for. 
further advances in the state-of-the-art in reporting techniques. 

An Analytical Framewprk for Forecasting 

The discussion thus far has focused on the political and economic 
factors affecting the evaluation and reporting needs of the 'LgAs and 
SEAS/ and has related some of those needs to the experiences of the 
Title I TACs in tSe develbpment of materials and provisior^ of assistance 
in the evaluation 6f Title u programs. While needs and experience 
provide two essential components for ,predicting evaluation trends, our 
future view can be somewhat enhanced by applying an analytical framework 
to the points we have discussed. The framework we have chosen is based 
on two assumptions: (1) that costs will play a major . role in determining 
the kinds of evaluation services SEAs and LEAs will employ and (2) that 
the information yield of evaluations will be a critical factor in 
determining their worth. ^ 

In Table 1, we have a comparative analysis of six types of evaluation 
we have identified as needed, in terms of their relative cost! and 
information yield for Chapters I and II of P.L. 97-35. 

■ 4t 



^The authors are indebted to Dr. Stephen L. Murray of the Northwest 
Regional Educational Laboratory for the suggested application of this 
framework. 
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Table 1 

Types of Evaluation vs. Cost and Information Payoff 



Needs 

Assessment 

Descriptive 
Evaluation 

Manageme^nt 
Evaluation 



Implementation 
Evaluation 



Program 

Impact Evaluation 

Cost Effectiveness 
Evaluation 



CHAPTER I 
RelatiV^e 
Cost per ' 
Stadept 



Low 



Low 



Low 



High 



Low 



Low 



CHAPTER II 
Relative 
Cost per 
Student 



Payoff 

Low High 

High Low 



Low Low 

(District) 

High 

(State) 

High \ High 

(District) 

Low 

(State) 



High 
High 



•4 

High 
High 



Payoff 
High 

High 



Low 

( District 

High 

(State) 

High 

(District) 
Low ' 
(State) 

Low 



Low 
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The rationale for our analysis of cost versus payoff for various 
types of evaluative data is based on two major factors, relative size of 
p*gram in terms of student enrolln^ent and potential utility. of the 
data. The 'following examples may serve to illustrate that rationale. 

Needs assessment!^ is required for both Chapter I and Chapter II- 

programs. Since the average size of Chapter I projects will be much 

larger than that of Chapter II projects, the relative cost per student 

will be lower for Chapter I. The payoff, however, in terms of the 

potential utility of the data, may be higher for Chapter II than ^ 
» 

Chapter I because multiple program demands for fewer dollars under 

'Chapter II create pressure for allocation (decisions to be^ata based. 

« 
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Descriptive evaluation data in the form of frequency counts, 
demographic statistics and other non-parametric data are relatively 
inexpensive to collect since they are often available as a by-product of 
administrative records systems. While the information they yield has 
limitations for statistical manipulation, it can provide decision makers 
with an abundance of quick, factual .data for responding to board 
inquiries and public in^ormatign needs and for analyzing trends. 

Management evaluation or the application of management consulting 
techniques to evaluative problems tends to be more useful in the higher 
order bureaucracy Where outcomes are less product oriented and more 
, emphasis is placed von support services and assistance. This is so 
because the techniques are less measurement based than some others and 
rely more on personal interaction and problem solving. 

Implementation evaluation is an example of a relatively high cost 
activity that can have a high payoff under the right conditions. It is 
mere appropriate for LEA than SEA applications since the LEA is the focus 
of program activity. A decision regarding whether to apply this approach 
or not might bear consideration if the program in question is based on a 
substantial resource commitment and is expected to yield strong results. 
In such cages, effective implementation takes on increased importance and 
the results of an implementation evaluation may make a critical 
, difference in terms of correcting program deficiencies before the program 
is termed a failure. 

Program impact evaluation will have a relatively low cost where 
program enrollment is high and a relatively high payoff where a 
predetermined need for the data has been established (e.g., a federal. 
State or local requirement). 
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Cost effectiveness evaluation will tend to have tha highest payoff 
where the total program outlay is largest. This is true, in part, 
because of the* vulnerability of the large program budget to political 
attack. Sinte Chapter I projects are larger, on the average than 
Chapter II, it is most likely to have a high payoff when applied to 
Chapter I. 

^OR TRENDS AND ISSUES IN EVALUATION , ' • 

' The literature on evaluation has exhibited ^ few major trends over 
the past several years. One of these trends has been toward increased 
utilization of evaluation. Another has involved increased use of so 
called "qualitative* methodology. We have also seen attempts to apply, 
methods from other disciplines such as journalism, phiy.osjophy and 
business, to educational evaluation. The Research On Evaluation Program 
at NWREL, under the direction of Dr. Nick Smith, has commissioned the 
development of evaluation metaphors from many disciplines. This has 
resulted in some innovative and useful perspectives on evaluation such, as 
Investigative Journalism (Gub^, 1981), cost analysis (Levin, 1981) and 
management consulting (Stanfield, 1981) as alternative approaches to 
evaluation. 

Evaluation Utilization 

Much of the recent literature has focused on the issue of evalua^tion 
utilization (Alkin, Daillak and White, 1979; Patton, 1978). In a recent 
review of the utilization literature (Hansen, Martin and Oxford, 1981) 
conducted under our Title I Evaluation TAC contracts at NWREL, we 
identified four major categories of factors affecting the utility of 
evaluations: (1) technical-methodological, (2) role related, 
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(3) cor.ununications, and (4) political factors. The first includes 
problems of design and instrumentation. The second deals with the roles 
of program managers, evaluators and others involved in or 'directly 
affected by the evaluation. The third deals with types of evaluative 
information and how they are communicated through the evaluation 
process. The fourth examines the political context as an essential and 
often neglected factor in all evaluation activities. 

The Quantitative vs. Quali tative Debate 

— ^^r;: ^ ^ 

The utilization issue is at the core of a continuing methodological 
debate that -has split the* evaluation community into <tewo opposing camps; 
one based on adherence to the time worn and revered experimental approach 
derived from psychology ^ and the other based on the field study or 
ethnographic approach of anthropology and sociology. The debate is 
often, albeit simplistically, characterized as "quantitative vs. 
qualitative," although this does injustice to both views^ 

Law < 19^80) calls for new approaches in three critical areas of 
evaluation: design, context and use. On the issue of design. Law 
recognizes the trend away from the quasi-experimental, psychometrically 
based evaluations of the 1960s and toward more qualitative "functional" 
evaluation designs as a healthy trend. He expresses concern that the 
pendulum might swing too far from a purely quantitative to a purely 
qualitative approach. He calls for "an appropriate balance" of 
methodology, based on a ca*reful determination of the information needs of 
the client (page 16). 

The quantitative position dominated the field throughout the 1960s 

and 1970s. More recently, through the works of such authors as Cuba 

(1972), Alkin, et. al (1979) 'and Rist (1980) and others, we have seen the 

• • 

14 

17 



emergence of a trend toward a greater reliance on qualitative methodology 
in educational evaluation. This trend has been perceived by many as the 
wave of the future and iias led to the prediction that there will be a 
decline in the number of evaluations focusing on achievement outcomes as 
measured by standa^rdized test scores, yielding to an increa^ in case 
studies and other (qualitative methods. In our view this prediction 
cannot be supported by our knowledge of the current political and 
economic conditions influencing evaluation policy. 

Management Consulting as Evaluation 

Jonathan Stanf ield^ (1981) a management donsultant, commissioned by 
NWREL's Research On Evaluation Project to conduct a management review of 
evaluation practice at th^ SEA level concudes that the paradigm for 
e<Jucational evaluation is changing from the "big study" research based , 
methodology born with the evaluation requirements of the Elementary and 
Secondary Education Act of 1965, to $n organization-based function 
supporting SEA management. Stanfield's rationale includes ttie \ 
observation that "evaluation studies" are often slow, time consuming 
affairs, which often produce ambiguous or legally challengeable results 
offering little in the way of policy guidance. Such studies- are , 
contrasted with continuous input to program management .tesulting from 
data base inquiries. Stanfield points \put that the time scale Tor policy 
decisions is compressed as the locus for decision shifts from the federal 
to state or local level. SEA and LEA administrators need data about 
their programs on a more immediate basis than does congress. One might 
infer a negative relationship between level of policy decision served, 
e.g., federal, state, local o]p jschool site and ipnediacy of need. 
According to Stanfield's analysis, the type of large scale evaluation of 



15 



the type funded in the 1960s and 1970s, designed to guide major social 
policy decisions, will diminj.sh, if not disappear entirely in the 
future. Stanfield developed his perspective in advance of the^ advent of 
P.L* 97-35, based mostly on discussions with key administrators^ in SEA 
poltcy setting positions. The occurrence of block grants in both 
education and other social service areas would seem to lend support to 

r % 

his views. 

^The field of management consulting offers the evaluator ways to 
provide directly relevant information to key decision makers on an 
ongoing basis. The management consultant funttions manner very 

similar to that which we advocate in providing technical assistance. A 
management consultant .will focus on a specific problem ared such as 
productivity, cost or communications and will work directly with decision 
makers in devel.oping solutions. 

2 

Accomplishment Auditing 

Increased cost consciousness and reduced funds for evaluation may 

result in an increase in the use of accomplishment auditing as an 

evaluation activity. With less money available for districts to use for 

* 

third party or contract evaluations, it may be necessary for program 
staff tq^ conduct, outcome evaluations themselves. The independent 

I 

accomplishment audit"* provides a low cost means of improving the quality 

'of these internal evaluations by means of an external review process. 

The audit serves two basic functions: 

1. The auditor's/critique of evaluation plans ap^ procedures 
strengthens the internal evaluation process. 



^Based on information provided by the ^ufdit and Evaluaton Program 
of the Northwest Regional Educational Labora|K)ry, Dr. Mark Greetie, 
Director. 
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2. The auditor's certification of the project evaluation reports 
lends credibility to^h^ repo]:ted outcomes. 

The accomplishment audit involves^a technical review of evaluation 

documents, including the evaluation plan and reports as well as onsite 

reviews of the data collection, tabulation, analysis and reporting 

activities. 

The convergence of the audit approach with ^he need f-or fiscal 
accountability coul^ xesult in the development of a new professional 
specialty, that of the auditor-analyst. This new specialist would have a 
good groutiding in both education and accounting. His/her function would 
be to provide legislators and other funding and policy bodies with 
information on the soundness of the fiscal data provided^ by the program 
manager .and the validxty of the' reported outcomes. - 

* * • 

Political Awareness . . • - %:^ . . 

- . . . . ' . V. ^- 

l*aw (1982) advocates increased .evaluator awareness of the political 
context of evaluations. EvaluaUors need to become more knowledgeable of 
the legislative process. They must learn to recogrtize th'at all 
legislation is born of compromise. They need to design* evaluation 
studies that go beyond the mere requirements of the law if they are to be 
of use to policy makers. Law pos^s three general evaluation questions/of 
interest to legislators: 

1. Are you doing what we told you to do? 

- 2. How we|.l are you doing it? 

3. What ^londitions exist now in program schools that are different 
f /omN)ef ore? 

Each question has a number of implications for program management to 
consider, e.g., is^the right population ng served; what ate examples 
of success/failure; is there differential impact? Through an 
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understalnding of the political ambience of a program, an evaluator can 
educate and lead' policy makers to a greater appreciation of the 
information evaluation can providg. 

* n 

THE sea's perspective 

In recent planning discussions ^ with our own SEA clientele we find 
that views are* divided on the SEA's evaluation policy under the new law 
and that Chapters I and II are indeed being approached quite 
differently, in most of the 14 states we serve through the Title I TAC, 
the SEA will continue to require or at least expect LEAs "to ^conduct 
Chapter I evaluations using the T^pe I Evaluation anti Reporting System! 
In a. few states t'h^ SEA will r/lax the requiremtent for '^separation of 
selection from the pretest with Model A. ^ The pr^vailing philosophy seems 
to be that TIERS is producing better quality data than was ever available 
before and now that the system is in place we might as wel^ keep it. 

Section 555 of Chapter I (paragraph b) requires the SEA to keep such 
records and provide such inforination to the Secretary as may be required 
for fiscal audit and program evaluation. A similar clause (Section 556, 
paragraph b) requires the LEA to provide assurance to the SEA that such 
rebords as might be needed foij fiscal auSit and program evaluation will 
be kept. Many SEA evaluators and federal projects coordinators ar* 
.interpreting these statements as an indication that the Secretary will 



request such data at some point in the future and that the most effective 
way to respond to such a request would be to continue to collect IIERS 
data. ^ • > 
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There is less clariby at SEA level regarding evaluation pqiicy under 
Chapter II. Drs. Nick Smith and Darrel Caulley of NWREL are at present 
conducting a study of SEA and LEA evaluation policy changes in response^ 
to Chapter II. Preliminary data from a telephone survey of eight SEA 
level evaluators and one large district in each of eight states seem to 
indicate a "wait and see" attitude being adopted. None of the 

I 

respondents had indicated any changes in evaluation policy toward 

' r ' 

Chapter II programs from that'l^hich preceded Chapter II. It is not at 
all clear what this means but^ the researchers believe that a state of 
confusion exists, at present, regarding, evaluation ppiicy for Chapter II. 

A, FUTURE FORECAST 
Given the views expressed in recent literature^^n evaluation trends, 
our own experiences with Title I and the views of practitioners regarding 
future SEA and LEA evaluation policy, we would now hazard a forecast of 
evaluation trends for the next three to five year period. Perhaps the 
most straightforward, albefit imprecise method of making our forecast 
' WouJldJpe to estimate the pijQbability of each of the six types of 
evaluation previously mentioned bei^ applied in a broad scale fashiA|i. 
Thes^ estimates, admittedly subjective, but based on our und'erstanding of 
^the factors currently influencing evaluation practice, are s^hown i.n 
.Table 2. 

The wQrds Hi^gh, Medium and Low are used to indicate, in a gross 
fashion, o^r "best guess" regarding the probability of applica-tion of 
each of the approaches listed. 



•^Personal communication. 
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Tabl 

Probability Assessment of types of Evaluat'ion 



Needs Assessment 
Descriptive Evaluation 
Management Evaluation 

Implementation Evaluation 

Program'^^Impact Evaluation 
Cost Effectiveness Evaluation 



CHAPTER I 
Probability of 
Application 

H^Sh . 

High 



Low (District) 
Low ((Estate) 

High (District) 



High 



Medium 



CHAPTER II 
Probability of 
Application 

High 

High 

Low (Di^i^ict 
Medium (St^te)« 

'Medilfm C^i^tr^ict) 
Low (State)^« 

High 

Low 



An q^mination of Table 2 in combination with Table 1 will reveal our 
logic in making these predictions. Beginning with needs assessment^, the 



case seems clear. Both Chapters I and II require it, therefore there is 
a very high probability pf .needs assessments being conducted, even though 
costs are relatively high and information payoff is low as shown in 
Table 1. 

Descriptive evaluation , in the form of compilation of demographic 
data, program <^scriptions, frequency counts and other data that are 
easily compiled, stored and retrieved constitutes a low-cost, high-payoff 
approach for both chapters, and will therefore be of a high probability. 
This is an area of great potential for the myriad of microcomputers now 
beginning to appear in schools, district and state offices. 

Management evaluation , including the management consulting approach 
to provide program managers and policy makers with information for 
on-line decision making will have a relatively low probability of being 
applied to Chapter I at LEA level and SEA level and a medium probability 
of application to Chapter II at the state level. As discussed earlier, 
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most SEAS plan very few changes in evaluation requirements under 
Chapter I from tho3e of Title I. This means jbhat evaluation studies 
other than those currently being done with TIERS type data would bear 
additional costs. Even though maijagement evaluation costs are relatively 
low compared to other approaches, any additional evaluation costs may be 
prohibitive in those states where the administrative set aside is being 
reduced. There may be a slightly higher payoff to SEAs to conduct these 
types of studies Under ChSpt^er II, where program structure is less well 
defined and priorities for project fundirfg may.be in conflict. 

Program impact evaluation will be required at both state and local 
level for both chapters, thus regardless of payoff, it will be conducted, 
although financial constraints may reduce the size and scope of such 
efforts. On larger projects with higher academic achievement as the 
goal, the TIERS or a similar process will likely be applied. Smaller 
projects and those with less well defined outcomes may use a variety of 
other approaches to ascertain effects, such as objectives checklists or 
.attitude measures. For the most part however, these will be i]i^imal 
efforts conducted at low cost. 

Cost effectiveness evaluation will, contrary to the rising tide of 
cost consciousness, not appreciate an overall high probability of 
applicati^on although it may receive relatively more attention under 
Chapter I than II. Cost effectiveness, as an approach to educational 
evaluation, remains an esoteric field, practiced by few and understood by 
even fewer. Levin (1981) and others have made substantive contributions 
to demystifying this area of endeavor and. have developed some 
straightforward procedures for its application, but there remains a ^reSt 
task to be accomplished in training evaluators in its use and in creating 
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an awareness of these procedures in the educational community. This is 
not to say that there w:^/l1 not be an increase in the attempts to collect 
cost data' and in some way link those data to outcomes-. We are seeing^ 
this occur on a frequent basis now. Application of cost effectiveness 
evaluation of ^ rigorous type on a broa<J scale will be slow in arriving, 
however. * ^ 

Qualitative Evaluation Methods Re-Examined 

The quantitative-qualitative debate continues. Five of the six types 
of evaluation we have identified as being most appropriate are basically 
■quantitative.' This reflects our view that although qualitative methods 
have much to offer, conditions are not propitious for their wide spread 
use in the near future. Evalu«itors around the country have been 
subjected to characterizations, sometimes unjustly, of "bean counters" or 
■fuzzies' depending on whether they viewed themselves as having a 
primarily quantitative or qualitative orientation. For the time being 
the bean coufiters dominate. This condition will likely continue until 
educational funding and policy constraints arp relaxed thus providing the 
resources and environmental conditions needed to encourage further 
development and application of qualitative approaches. 

We have heard the thunder of the pending storm that threatens to 
deluge us with ethnographic evaluations. To date, however, we have only 
receiyed a light sprinkle. Where is the deluge? What appeared a few 
years ago to be the evaluation Zeitgeist, now appears as less than the 
'true spirit. Qualitative evaluations, generall^ ^speaking* but with noted 
exceptions, tend to be costly* When evalution tunds were in good supply 
there was cause for hope that ethnography and other non-psychometric 
approaches to evaluation would gain wider acceptability. Now, however, 
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in an 6ifa of fiscal conservatism, the costs of conducting an evaluation 
study itiust stand the chalTenge of comparison against the potential for 
effecting an improvement in a social program. This is a very^^difficult 
challenge aqd the (labor) intensive ethnography cannot, except In rare 
cases, meet the challenge. Other types of qualitative studies such as 
the management consultant approach or implementation 'evaluation have a 
muph better chance of succeeding since they have as intended outcomes the 
improvement of the educational process at only a sdight cost to the 
project. This is especially true iR the case of management evaluation, 
where there is very little new or original data collection required. 

SUMMARY AND CONQLUSIONS 
Many recent changes in educational policy have influenced the types 
of evaluation we can expect to see practiced in the near future. The 
primary channels through which these changes are occurring are 
legislative and economic. Legislatively, the major influence is the 
Education Consolidation and Improvement Act of 1981. Economically, LEAs 
and SEAS are caught in a bind between local initiatives to reduce taxes 

* m. 

and rising inflation. 

Evaluation as aj^iscipline has been severely criticized for its 
failure to be as effective a force for educational improvement as many 
had hoped. Over time, certain trends have emerged in, evaluation 
resulting in an increased tendency toward qualitative methodology and 
away from the strong roots of experimentalism and quantitative 
methoddlogy. The political, philosophical, legislative and economic 
factors of recent years have slowed, if not stopped the, qualitative trend. 
In this paper we have reviewed the block grant legislation for 'the 
^ evaluation needs and implications it presents for both LEAs and SEAs. V/e 
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have applied our own experience in the areas of evaluation, technical 
assistance and research to draw inferences about the types of evaluation 
activities SEAs and LEAs will be most likely to need. By examining all 
of these factors, we have arrived ^at the following conclusions regarding 
evaluation policy and practice during the 1980 's. 

1. Becaus^CK)f reduced funding levels and lacif of policy support fc^ 
V evaluation, large scale sumroative evaluations of the type 

conducted in the 1960s and early 1970s will not be conducted on 
the same scale. 

2. ^ The major factor influencing evaluation policy and practice in 

the 1980s will be economics. Evaluation methods that offer high 
information pay off at relatively low cost will increase in 
use. This may result in ah increase of management oriejited 
studies based on techniques derived from management consulting. 
In this same vein, educational accomplishment audits may 
increase in popularity relative to third party evaluations*. A 
combination of outcome audit and financial audit could be a 
logical result of the need for fiscal accountability and ootcome 
validation. This could result in a new role for the evaluator, 
that of auditor-analyst. 

3. Qualitative approaches to evaluation such as ethnography, case 
studies and other methodologies involving intensive use of 
skilled professional labor for extended periods of time, will 
not be^ widely used due to their rielatively high costs. This is 
likely^ despite the long and widely acclaimed trend toward the 
use of such methodology. 

4. ^he requirement for program impact evaluation remains for both 
Chapters I and II of ECIA-81. The n^w law, however, provides 
fewer prescriptive requirements and affords greater flexibility 
in the ways in which programs are evaluated and data are 
repotted by LEAs and SEAs. In those SEAs and LEAs where 

* evaluation systems are well established and seem to be working - 
effectively, few changes in methods and reporting procedures 
. will occur. In states where th^re has been a strong "compliance 
orientation" the relaxing of requirements and weakened 
regulations will result in token evaluation efforts. 

5» Finally, although not dealt with extensively in this paper, 
technology will play an increasingly important role in 
Valuation activities. The microcomputer has brought powerful 
data manipulation capabilities within reach of virtually every 
school bailding. As teachers, evaluators and administrators i 
seek ways to make the best use of this technology, we may see a 
quiet revolution take place in the application of microcomputers 
in evaluation. 
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